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ABSTRACT 

Characteristics of university courses and student engagement were 
examined in relation to student ratings of instruction. The Universal 
Student Ratings of Instruction instrument was administered to students 
at the end of every course at a major Canadian university over a three- 
year period. Using a two-step analytic procedure, a latent variable path 
model was created. The model showed a moderate fit to the data (Com- 
parative Fit Index = .88), converged in 10 iterations, with a standard- 
ized residual mean error of .03, yi (149) = 1988.59, p < .05. The model 
indicated that course characteristics such as status and description are 
not directly related to student ratings. Rather, they are mediated by 
student engagement, which is measured by student attendance and 
expected grade. It was concluded that, although the model is statisti- 
cally adequate, many other factors determine how students rate their 
instructors. 


RESUME 

Cette recherche evalue les caracteristiques des cours universitaires 
ainsi que la participation des etudiants en rapport avec T evaluation 
estudiantine de l’enseignement. Pendant trois ans, dans une grande 
universite du Canada, on a administre un outil devaluation appele 
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« Universal Student Ratings of Instruction » [evaluation estudiantine 
globale de l’enseignement] aupres des etudiants, a la fin de chaque 
cours. On crea un modele d’analyse a variables latentes en utilisant une 
methode d’analyse a deux etapes. Le modele s’accordait passablement 
aux donnees (Indice d’ajustement comparatif = .88), a converge dans 
dix iterations avec une deviation standard a residu normalise de .03, / 2 
(149) = 1988.59, p < .05. Le modele demontre que les caracteristiques 
des cours tels que le statut et la description ne sont pas directement 
lies aux evaluations des professeurs. Elies sont plutot influencees par 
la participation des etudiants, et mesurees par l’assiduite des etudiants 
ainsi que par les notes anticipees. Bien que le modele soit adequat 
statistiquement, les resultats suggerent qu’il existe plusieurs autres 
elements influen^ant Revaluation des professeurs par les etudiants. 


INTRODUCTION 

Since student evaluations of teaching effectiveness can have a significant 
impact on instructors’ careers (Sprinkle, 2008), considerable research has been 
conducted to ensure that these ratings are valid. Greenwald (1997) concluded 
that these studies “give a clear impression that major questions of the 1970s 
about ratings validity were effectively answered and largely put to rest by 
subsequent research” (p. 1184). Nevertheless, researchers suggest that various 
characteristics about the courses and the students themselves may, in part, con- 
tribute to these ratings. Course and student characteristics have typically been 
studied for their impact on student ratings of instruction, while studies examin- 
ing the relative importance of several of these characteristics simultaneously 
are rare. Accordingly, in the present study, we set out to determine the extent 
to which characteristics about the course and student engagement in the course 
influence how students rate their instructors. 

Validity 

Validity refers to the extent to which a measure accurately quantifies the 
construct being measured (Messick, 1989). In the context of teaching evalua- 
tion, when student ratings reflect the process of instruction (i.e., what teach- 
ers do when they teach) and the impact of teachers on the desired products 
of instruction (i.e., student learning), the ratings are said to be valid (Abrami, 
d’Apollonia, & Cohen, 1990). Over the past several decades, a considerable body 
of research, commentary, and criticism has been focused on issues related to the 
validity of student ratings (Abrami, 2001; Oiy & Ryan, 2001). Having explored 
historical trends on the validity of student ratings, Greenwald (1997) reported 
that over a 25-year period beginning in the 1970s, more publications favoured 
evidence for, rather than against, validity. These conclusions about the adequate 
validity of student ratings have been supported by other reviewers (Cohen, 
1981; McKeachie, 1979; Murray, 1984). 
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Although teaching effectiveness is one factor that determines student rat- 
ings, additional factors outside of teaching, such as the characteristics of the 
courses being evaluated and the engagement of students conducting the evalu- 
ations, may influence how students rate their instructors. 

Course Characteristics 

University and college courses vary according to several characteristics. 
Courses can be described by their type and length. Types of courses typically 
provided in higher education include lectures, labs, practica, tutorials, and dis- 
tance learning. Little research has examined differences in student ratings ac- 
cording to these types of courses, but one study found that labs received higher 
ratings than lectures or tutorials (Beran & Violato, 2005). It is possible that 
hands-on application of theory and research leads to greater satisfaction in 
learning, and, hence, higher student ratings for instruction. In terms of course 
length, little research has examined whether longer courses receive higher rat- 
ings than shorter courses. In a preliminary study by Beran and Violato (2005), 
course duration was related to student ratings, but the relationship was small 
(effect size— Cohen’s d— of .15). Given the greater opportunity for contact and 
learning from an instructor in a full-year rather than a half-year course, stu- 
dents may gain more knowledge and skills, which, in turn, may yield higher 
student ratings. 

Courses also vary according to their status, that is, whether they are re- 
quired, are within the student’s program, and have a heavy workload. Arreola 
(1995) and McKeachie (1979) each examined the role of course requirement and 
found that student ratings were lower for required courses, compared to elec- 
tive courses. Similarly, students may give higher ratings to courses outside their 
area of study than to courses within their program if those courses are taken 
for interest. Theall and Franklin (2001) also suggested that courses within the 
student’s program area tend to be given low ratings. Perhaps the opportunity 
to select elective courses and courses outside their program provides students 
with options rather than requirements. Moreover, those courses offered outside 
of student programs and selected as electives may have a lower workload re- 
quirement. 

Empirical evidence of this relationship between workload and student rat- 
ings is mixed, however. Some research suggests a positive and direct relation- 
ship between workload and student ratings (Marsh & Roche, 2000), perhaps 
because it presents a greater challenge and a sense of value in learning for the 
investment of time and effort. Students in these courses may also feel that they 
have learned more and, thus, give high ratings. Other studies, however, report 
that higher workload is related to lower student ratings (Greenwald, 2002; Ku- 
lik, 2001); if the course workload seems excessive and causes students a high 
degree of stress, it may result in low ratings. 
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Student Engagement 

Student engagement is a multi-dimensional construct that includes active 
learning and collaboration (National Survey of Student Engagement, 2005). 
When students participate in class discussions and presentations, as well as 
converse with their instructors, they can develop and enrich their knowledge 
and skills (Brint, Cantwell, & Hanneman, 2008). These activities occur when 
students attend class regularly and are likely to create high expectations for 
good marks. These student-engagement behaviours can also lead to enhanced 
learning, which, in turn, may result in positive student ratings for the course. 
Indeed, Greenwald (2002) concluded that students who receive high marks are 
likely to give high ratings. Student attendance as another student-engagement 
characteristic that may influence student ratings has not been studied exten- 
sively, however. Students who attend classes frequently may be more motivated 
and interested in the course and may rate the teaching effectiveness as high, 
compared to students who attend only sporadically. Beran and Violato (2005), 
for example, found this to be the case. 

Various course characteristics and student engagement may relate to stu- 
dent ratings of instructors. Course descriptions such as type and length may 
influence ratings; for instance, applied courses such as labs and long courses 
may receive high ratings. Course status may also affect ratings. That is, elective 
courses, courses outside the student’s program of study, and courses requir- 
ing a moderate workload are likely to receive high student ratings. Given the 
small and sometimes mixed results of previous research on the relationship 
between course characteristics and student ratings, student engagement was 
introduced into the present study to determine its relative importance to student 
ratings. When students are actively involved in their learning, as demonstrated 
by frequent class attendance and high expectations for marks, they should give 
their instructors high ratings. The main purpose of the present study, therefore, 
was to test a latent variable path analysis (LVPA) model that integrates several 
course characteristics, student engagement, and student ratings of instruction. 
An advantage of applying LVPA to this problem is that it allowed us to identify 
and measure several latent variables and determine their interrelationships in a 
full model of student ratings of instruction. 

METHODOLOGY 
Sample and Procedure 

A sample of 371,131 student ratings across all faculties at a major Canadian 
university over a three-year period (from the 1999 to the 2002 winter term) was 
obtained. Ratings were conducted at the end of every course, session, and term, 
with the majority of responses obtained during the fall term (i.e., December), 
followed by the winter term (i.e., April), which is consistent with student enrol- 
ment. The average response rate was 61%. 
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Instrument 

Student ratings were derived with the Universal Student Ratings of Instruc- 
tion (USRI) instrument, which consists of 12 items that were constructed based 
on other published student-rating measures used in research (e.g., Marsh, 1991). 
The responses provided in Table 1 were coded on a 7-point response scale, 
ranging from strongly disagree to strongly agree, with a higher score indicat- 
ing a more positive rating. The internal consistency alpha reliability coefficient 
of the 12 items was .92, indicating that the scale is internally consistent. The 
structure of the USRI has been examined in previous research and is consid- 
ered to be a unidimensional measure of teaching (Beran & Violato, 2005) that 
is consistent with other measures of teaching effectiveness (e.g., Greenwald & 
Gillmore, 1997). To ensure anonymity, student identification numbers were not 
recorded; therefore, the number of individual students who completed the rat- 
ings is unknown. 

Table 1 

Mean and Standard Deviation of the USRI items (n = 371,131) 



M 

SD 

The overall quality of instruction 

5.65 

1.24 

Student questions and comments were responded to appropriately 

5.94 

1.16 

The course content was communicated with enthusiasm 

6.00 

1.07 

Students were treated respectfully 

5.83 

1.32 

Opportunities for course assistance were available 

6.04 

1.22 

The course outline or other descriptive information provided enough 
detail about the course 

6.07 

1.24 

The course as delivered followed the outline and other course descriptive 
information 

5.96 

1.15 

The course material was presented in a well-organized manner 

6.24 

1.08 

The evaluation methods used for determining the course grade were fair 

5.72 

1.38 

Students' work was graded in a reasonable amount of time 

6.01 

1.19 

I learned a lot in this course 

5.76 

1.37 

The support materials used in this course helped me to learn 

5.61 

1.39 


In addition to rating their courses, students were asked about the character- 
istics of each course, specifically, its type ( 1 = lecture, 2 = lab, 3 = practicum, 4 
= tutorial, 5 = distance), its duration (0 = half year, 1 = full year), and its status, 
that is, whether the course was in their program of study ( 1 = in department, 2 = 
not in department, 3 = unknown), whether it was required (1 = required course, 
2 = required choice, 3 = elective), and how its workload compared to similar 
courses ( 1 = much lower, 2 = lower, 3 = same, 4 = higher, 5 = much higher). 
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In terms of their engagement with the course, students were asked to in- 
dicate their attendance rate for each course they were rating (1 = 0-20%; 2 = 
21-40%; 3 = 41-60%; 4 = 61-80%; 5 = 81-100%) and the grade they expected 
to receive at the end of the course (11 = A, 10 = A-, 9 = B+, 8 = B, 7 = B-, 6 
= C+, 5 = C, 4 = C-, 3 = D+, 2 = D, 1 = F). These characteristics are shown in 
Table 2. 


RESULTS 

When analyses of variance were conducted to examine student ratings 
against the variables listed in Table 2, the resulting effect sizes (r|p 2 ) were all low 
(.00 to.07). Thus, according to these characteristics, the ratings did not differ to 
a large extent. To further explore the relationships among course characteristics, 
student engagement, and student ratings, a two-step analytic procedure using 
two separate random subsamples of 2,000 was conducted. In the first step, princi- 
pal component extraction with varimax rotation was used. Three factors emerged 
after five iterations, explaining 55.9 % of the variance. The first factor, course 
status, which accounted for 23% of this variance, was measured by workload 
(factor loading = .54), course within department (factor loading = .78), and course 
required (factor loading = .82). The second factor accounted for 16% of the vari- 
ance and reflected course description, with factor loadings of .67 for course type 
and .74 for duration. The third factor, student engagement, accounted for 17% of 
the variance and was measured by attendance (.75) and expected grade (.75). 

These three factors were then used to derive latent variables in the latent 
variable path analysis, using a second randomly drawn sample of 2,000 (see 
Figure 1). The model was re-specified until the best statistical fit was obtained, 
following the principle of parsimony. The model converged in nine iterations 
and provided a moderate fit for the data, x 2 (149) = 2304.40, p < .05; standard- 
ized root mean square residual (SRMR) = .03 ; and comparative fit index (CFI) = 
.86. Thus, the model accounted for 86% of the variance-covariance in the data. 
There was a range of residual coefficients: .57 to .84 for the 12 USRI items; 
.70 to .97 for course status; .84 to .85 for course description; and .82 to .97 for 
student engagement. 

As shown in Figure 1, student ratings differed according to student engage- 
ment and course characteristics. Student ratings are depicted on the right side 
of the model, as measured by all 12 USRI items. The latent variable “student 
engagement” was measured by the grade students expected to receive in the 
course and their frequency of attending the course. This variable was directly 
related to student ratings— students expecting high grades and attending class 
on a regular basis provided high ratings for the course. Two types of course 
characteristics are shown on the left side of the model. Course description was 
measured by length and type. This variable was related to course status, mea- 
sured by course requirement, department status, and workload. Although all 
of these course variables influenced student engagement unidirectionally, they 
were not directly related to student ratings. Thus, student engagement mediated 
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Table 2 

Course Characteristics and Student Engagement (n = 371,131) 


Characteristics 

n* 

Percentage 

Student 

Grade 

54,148 

14.6 

A 

75,875 

20.4 

A- 

78,251 

21.1 

B 

70,619 

19.0 

B‘ 

33,309 

9.0 

C + 

16,930 

4.6 

C 

13,939 

3.8 

c- 

5,446 

1.5 

D + 

1,174 

0.3 

D 

1,040 

0.3 

F 

361 

0.1 

Missing 

20,039 

5.3 

Attendance 



0-20 

769 

0.2 

21-40 

1,163 

0.3 

41-60 

4,816 

1.3 

61-80 

27,904 

7.5 

81-100 

332,847 

89.7 

Missing 

3,632 

1.0 

Course 

Length 



Half 

352,209 

94.9 

Full 

18,922 

5.1 

Type 



Lecture 

330,927 

89.2 

Lab 

7,510 

2.0 

Practicum 

1,097 

0.3 

Tutorial 

5,232 

1.4 

Distance 

2,839 

0.8 

Other 

3,717 

1.0 

Missing 

19,809 

5.3 

Workload 



Much lower 

6,457 

1.7 

Lower 

37,997 

10.2 

Same 

209,947 

56.6 

Higher 

87,164 

23.5 

Much higher 

27,057 

7.3 

Missing 

2,509 

0.7 

Program 



In department 

218,121 

58.8 

Not in department 

126,417 

34.1 

Department unknown 

23,116 

6.2 

Missing 

3,477 

0.9 

Required 



Required course 

190,102 

51.2 

Required choice 

80,054 

21.6 

Elective 

98,613 

26.6 

Missing 

2,362 

0.6 


Note.' number of responses 
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Note. Coefficient loadings for the 12 USRI items on Student Ratings range from .57 to .84. 

Figure 1. Latent variable path model of student ratings employing Maximum Likeli- 
hood estimation ( n = 2,000). 


the effect of course description and course status on student ratings. However, 
the residual variances for student engagement and student ratings were high 
(.96 and .87, respectively). In other words, course description, course status, and 
student engagement were significant in the model, but there was considerable 
residual variance in student ratings. The same is true for the residual variance 
in student engagement. This result is consistent with the low effect sizes from 
the ANOVA analyses, which showed that student ratings did not differ greatly 
according to course characteristics and student engagement. 

DISCUSSION 

The main finding in the present latent variable path model is that course 
characteristics (course description and course status) are indirectly related to 
student ratings since they are mediated through student engagement, which 
directly influences student ratings. However, the large residual variances and 
modest model fit suggest that these characteristics explain student ratings to a 
limited extent only. 


T. Beran and C. Violato / Student Ratings of Teaching Effectiveness 


9 


Course Characteristics 

Course characteristics were not directly related to student ratings in the 
model. Thus, ratings did not vary as a direct function of the length and type 
of course; instead, the influence of the latter was mediated through student 
engagement. Although shorter courses provide fewer opportunities for learning 
than longer courses, learning may be condensed and students may be motivated 
to work harder over a shorter duration, compared to longer courses, which have 
more opportunities to procrastinate (Schraw, Wadkins, & Olafson, 2007). Also, 
compared to lecture-type courses, applied courses (e.g., labs, tutorial) did not 
receive higher ratings in the present study. Some research has shown that the 
invitation for student response within a lecture-based class increases learning 
(Blood & Neel, 2008). Thus, rather than measure differences across courses ac- 
cording to their type, future research should assess instructors’ teaching styles 
within courses in relation to student ratings and student engagement. Our study 
may not have shown differences in student ratings across courses because these 
influences are felt only when mediated through student engagement. 

The present study also showed that course-status characteristics are not di- 
rectly related to student ratings but are mediated through student engagement. 
Courses outside the student’s program of study and those taken as electives do 
not necessarily result in higher student ratings. Presumably, courses taken with- 
in a program area are meaningful because students selected their program of 
study. Thus, students may have similar interests in program courses, compared 
to out-of-program courses. Also, within programs there is often flexibility in 
choosing courses or course sections. 

In terms of course workload, some previous research has shown that stu- 
dents give high ratings for high-load courses (Marsh & Roche, 2000) ; however, 
other research has shown that they give low ratings for high-load courses (Gre- 
enwald, 2002; Kulik, 2001). This was not the case in the present study (we found 
a small effect size). The discrepancies in the results from previous research and 
the small effect size in our study suggest that workload may be minimally re- 
lated to student ratings and that other factors may mediate this relationship. 

Student Engagement 

Student engagement, as measured by attendance and expected grade, was 
related to student ratings in the model. Accordingly, students with frequent 
attendance and high grade expectations give course instructors high ratings. 
When students are motivated and interested in the course, they are likely to 
participate in classes and miss few of them. By asking and answering questions, 
discussing concepts, sharing examples, evaluating ideas, and applying knowl- 
edge, students are likely to develop mastery in their learning. Such accomplish- 
ment will likely create expectations for high marks. 

Course status and description also predicted student engagement. Their en- 
gagement was higher in longer and more applied courses, as well as in those 
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courses that were outside their department, were electives, and had a higher 
workload. Perhaps greater interest in these courses increased the students’ en- 
gagement in their learning. Indeed, courses that students selected and that were 
applied, longer, and required more work were likely to involve students more in 
learning about the subject. 

Although the path coefficients suggest that student engagement mediates 
the relationship between course characteristics and student ratings, the residual 
variances of student engagement and student ratings were high. This suggests 
that ratings are determined to only a limited extent by student engagement. 
Similarly, student engagement is not largely determined by course character- 
istics. Thus, students who are not actively engaged in their learning may not 
necessarily give lower ratings to their instructors. Also, they may be engaged in 
all types of courses regardless of their characteristics. Perhaps students who find 
the class entertaining and informative without having to put effort into learn- 
ing give positive ratings. Additionally, students who are interested in a particu- 
lar topic may be actively engaged regardless of how the course is offered. 

Other Factors 

The present model suggests that factors other than course characteristics 
and student engagement determine how students rate their instructors. It is like- 
ly that characteristics about the instructor have more to do with effective teach- 
ing. Although a specific definition of effective teaching has not been identified 
(Goodwin & Stevens, 1993; Johnson & Ryan, 2000; Kulik, 2001), many descrip- 
tions focus almost exclusively on the instructional process (e.g., preparation of 
material, content knowledge). Arreola (1984), for example, regarded teaching 
as encompassing three broad dimensions: content expertise, instructional de- 
livery skills, and instructional design skills. These dimensions relate to the in- 
structional process as they reflect the skills and characteristics that promote or 
facilitate student learning. Lowman (1984) specified effective teaching accord- 
ing to intellectual excitement, which encompasses clarity and presentation of 
current materials, and to interpersonal rapport, which includes showing interest 
in students as individuals, encouraging creative and independent thought, and 
being warm, open, predictable, and student-oriented. Although not measured 
in the present study, these factors are likely stronger determinants of student 
ratings than student engagement and course characteristics. 

Implications for Future Research 

The present study has implications for future research. The other factors 
discussed above need to be examined in relation to student ratings, as well as in 
comparison to student and course characteristics, to determine their relative im- 
portance. Also, alternative measures of student ratings should be employed. The 
USRI instrument is unidimensional, and it is possible that multi-dimensional 
ratings will yield different results. Moreover, the residual variances of the latent 
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variables in our model indicate that these course and status descriptions are not 
good representations of the various types of courses taken by students. 

Summary 

Most researchers agree that teaching effectiveness may be defined as the 
degree to which an instructor facilitates student achievement (McKeachie, 
1979). Despite instructors’ anecdotal concerns about the potential influence of 
various factors outside their control (i.e., course characteristics and student en- 
gagement) on the amount that students learn, our study suggests that these play 
a small role in the ratings students give their instructors. Future research may 
well employ teacher characteristics and incorporate these into our proposed 
LVPA model, which includes student engagement as a latent variable. * 
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